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Abstract. The MaxEnt solutions are shown to display a variety of behaviors (beyond the traditional and 
customary exponential one) if adequate dynamical information is inserted into the concomitant entropic- 
variational principle. In particular, we show both theoretically and numerically that power laws and power 
laws with exponential cut-offs emerge as equilibrium densities in proportional and other dynamics. 

PACS. 89.70.Cf Entropy and other measures of information - 05.90.+m Other topics in statistical physics, 
thermodynamics, and nonlinear dynamical systems - 89. 75. Da Systems obeying scaling laws - 89.75.-k 
Complex systems 



1 Introduction 

The principle of maximum entropy is a fundamental idea 
of contemporary science. It states that, subject to known 
constraints, the probability distribution which best repre- 
sents the current state of knowledge is the one with largest 
entropy |H2j . In other words, let some testable informa- 
tion about a probability distribution function be given and 
consider the set of all trial probability distributions that 
encode this information. The probability distribution that 
maximizes the information entropy should be regarded as 
the optimal probability distribution with respect to the a 
priori available information. In most practical cases, the 
testable information is given by a set of conserved quanti- 
ties (average values of some moment functions), associated 
with the probability distribution in question. This is the 
way the maximum entropy principle is most often used in 
statistical thermodynamics. Another possibility is to pre- 
scribe some symmetries of the probability distribution. An 
equivalence between the conserved quantities and corre- 
sponding symmetry groups implies the same level of equiv- 
alence for both these two ways of specifying the testable 
information in the maximum entropy method. The max- 
imum entropy principle is also needed to guarantee the 
uniqueness and consistency of probability assignments ob- 
tained by different methods, statistical mechanics and log- 
ical inference in particular. The maximum entropy princi- 
ple makes explicit our freedom in using different forms of 

a alberto.hernando@irsamc.ups-tlse.fr 
b plastino@fisica.unlp.edu.ar 
c plastino@sinectis.com.ar 



prior information. As a special uniform prior prob- 

ability density (Laplace's principle of indifference) may 
be adopted. Thus, the maximum entropy principle is not 
just an alternative to the methods of inference of classical 
statistics, but it is an important conceptual generalization 
of those methods [2]- 

In this communication we reveal how to accommodate dy- 
namical information into the principle via a special treat- 
ment of the equations of motion that considers propor- 
tional and larger-than-proportional dynamics due to their 
importance in the study of complex systems. Some rather 
surprising results ensue (power-laws and power-laws with 
exponential cutoffs [3]) that illustrate the power of the 
approach. 

We demonstrate that taking into account dynamical infor- 
mation within MaxEnt involves adding to the pertinent 
Hamiltonian new terms and that these resemble the so- 
called "information cost" lucidly introduced by the authors 
of Ref. In this way we explicitly reconcile two appar- 
ent different viewpoints, i.e., that of the growth models of 
Simon [5] and the information-treatment of Mandelbrot 
[6] , showing that the equilibrium density of a growth pro- 
cess is the one that maximizes the entropy associated to 
the enlarged Hamiltonian introduced here. 

Our presentation is organized as follows: in Section 2 we 
present the basics of the problem; in Section 3 we describe 
the theoretical approach finding the equilibrium densities 
by means of MaxEnt according to the dynamical equation 
that governs the system at habd; in Section 4 we confirm 
our findings by means of numerical experiments with ran- 
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dom walkers, and we close drawing some conclusions in 
Section 5. 



2 Preliminary matters 

Let us define 

i) N as the total number of elements/members of a dis- 

crete set, 

ii) n c as the total number of special subsets into which 
the A^— elements can be grouped, 

iii) Xi as the number of members of the z-th subset, 

iv) n(x) as the total number of subsets with exactly x 
members. 

Considering that n(x) is a discrete distribution, the con- 
servation of both N and n c guarantees 



i) n(x) = n c , and 

x=l 

oo 

ii) 2_j xn ( x ) = N- 



Let us now consider the continuous limit of the distribu- 
tion n(x)/n c — > p(x)dx. 

Our goal here is that of finding out, via MaxEnt, the ex- 
plicit form of p(x)dx from either some simple expectation- 
values' constraints or, and this is the novelty, from dynam- 
ical information not of that kind. 



3 Theoretical approach 

3.1 Brownian motion: the ideal gas 

We first consider, as a control case, the dynamics behind 
the evolution of subsets of sizes x via the linear equation: 

x = k (1) 

where k involves a Wiener process, i.e., (ki(t)kj(t')) — 
KSijS(t — t') being K the variance of k. We are in the 
presence of the well-know Brownian motion, which obeys 
the diffusion equation 



d t p{x,t) = Ddlp(x,t), 



(2) 



with D the diffusion coefficient and p(x, t) the non-equili- 
brium density at instant t. Starting at t = with a Dirac- 
delta distribution p{x, 0) = 8{x — x') 1 the solution to this 
equation is a gaussian distribution of the form (we set for 
simplicity x' — 0) 



p{x,t) 



1 



_ Ml 
-e iDt 



(3) 



The Shannon entropy measure is defined (up to a con- 
stant) as 



S=- 



dxp(x) \ogp(x), 



(4) 



where fl is a "volume" in a;— space defined by a lower and 
an upper limit of allowed sizes x Q and Xm, respectively 
(.To < x < xm, thusly fl — xm — %o,)- As our first trial 
we use just MaxEnt with a normalization constraint 



/i / dxp(x) 



0, 



(5) 



where /x is the associated Lagrange multiplier. The density 
that extremizes this quantity is the constant one p(x)dx — 
Z^ 1 dx with Z = f2 the normalization constant (and also 
by definition the partition function [5]), representing the 
classical ideal gas — an ensemble of non-interacting par- 
ticles at constant density and gaussian distribution of ve- 
locities. As a second trial we add a constraint on the first 
moment (the mean value of the sizes) (x) — N/n c , 



S — jJL I dxp(x) 



X I dxp(x)x 



= 0. 



(6) 



The concomitant distribution is the well-know exponential 
density 

I \,i exp(-Xx)dx 
p(x)dx = , (7) 

followed by an ideal gas in the gravitational field. Note 
that xm could diverge here, but Xq has a lower bound. 
The constants Z and A = xq\ (defined for convenience) 
can be obtained according to 



Z = x - 



A 



A- 1 + 1 = 



AT 
n c x 



(8) 



1, AT 



We depict in Fig. 1 an arbitrary case for Xq 
250000 and n c = 100000 (A = 1/24). 
Let us assume that our constraint is a mean-energy-one. 
Then, the distribution ^ can be associated to a "Hamil- 
tonian" [2] 

H = Ax/x , (9) 

and is, for instance, the distribution followed by an ideal 
gas in the gravitational field. Having a Hamiltonian, it is 
straightforward to introduce a temperature here by mul- 
tiplying it by P = 1/T. The partition function defined as 
Z = J dx exp (— j3 H) [Eq. (|Sl) ] seemingly remains invari- 
ant but with a redefined A that changes in the fashion 
A -»• /3A, 



3.2 Geometric Brownian motion: the scale-free ideal 
gas 

It is well-known that some social and economic systems 
display a scale-free behavior [see, for instance, |7 |8|91 . and 
references therein]. Thus, we attempt now introducing a 
proportional growth into the orthodox Jaynes-MaxEnt treat- 
ment. We start by considering the equation, 



kx 



(10) 
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Fig. 1. Equilibrium density for the q = case (see Eq. (21 1 in 
the text) for x = 1, N = 250000 and n c = 100000 {A = lj2i). 
Black line: prediction from the MaxEnt, blue dots: histogram 
of the asymptotic distribution of brownian walkers. The shad- 
owed area in the inset around the line reflects the numerical 
error of the simulation 0(l/y/n^). 



where k is again indicative of a Wiener process, and, of 
course, we deal here with the very the definition of ge- 
ometric Brownian motion. We now proceed to linearize 
the dynamic equation via the new variable u = log{x/xo) 
obtaining thereby 

ii = k. (11) 

In analogy with the precedent case, we have the usual 
Brownian motion for the variable u obeying the diffusion 
equation. As for the x observable one has 



d t p(x, t) = Dd x {xd x (xp(x, t))) , 



(12) 



solved (with an initial Dirac-delta p(x, 0) = S(x — x')) via 
the log-normal distribution (we set for simplicity x' = 0) 



p(x,t) 



1 



i°g z (x) 

-e iDt . 



VAttDIx 

The Shannon entropy in the variable u is written as 



S = - 



dup{u) log p(u), 



(13) 



(14) 



with u instead of x. The first constraint is expressed, as 
usual, as 



S — jit / dup(u) 
n 



= 0, 



(15) 



that yields a constant density for u as p(u)du = Z~ 1 du, 
with Z = J? = log(xM /xq). Changing back to the observ- 
able x = xoe u we find 



p{x)dx 



1 dx 
Z x ' 



(16) 



which follows the density-behavior of the scale-free ideal 
gas (SFIG), as found before by means of Fisher's informa- 
tion in references [7)8|9j . 



Fig. 2. Same as Fig. 1 for the q — 1 case of Eq. (21 1 



We remind the reader here of Benford's law (BL) [10111 j . 
also called the first-digit law. As shown in Ref. [TT] by 
means of a bright heuristic derivation, the distribution 
that originates BL for first digits has the form of Eq. [16] 
Its occurrence is typical of low self-correlated data with 
no characteristic size, and thus agrees with the SFIG- 
definition of a non-interacting system with scale invari- 
ance. 

The second constraint is now expressed in terms of u via 
(x) = xo(e u ) = N/n c , and the Jaynes-MaxEnt extremiza- 
tion problem becomes 



S — /i / dup(u) — A I dup(u)e v 
J n 



0. 



(17) 



where we have used the definition A = xoX. We obtain 
the density p(u)du = Z^ 1 cxp(— Ae u )du. Changing back 
to the observable x we arrive at 



p(x)dx 



1 exp (—Ax/xq) 
Z x 



dx. 



(18) 



The pertinent constants are obtained from the constraints 
in the fashion 

Z = r(o,A) 

e- A _ N (19) 



AT(0,A) u c xq 

where r(a, z) is the so-called incomplete Gamma function. 
This is then the expected equilibrium distribution for an 
scale-free system, as those of opinion cluster dynamics in 
networks with fixed number of nodes as well as the 
number clusters n c [9]. We display in Fig. 2 the case for 
the same values of Xq , A^ and n c as in the preceding Section 
(now with A = 0.360743). 

Assume again that our constrain is a mean -energy-one . 
We can associate then to the distribution ( 18 ) the effective 
proportional-growth Hamiltonian [2] 



H = Ax/xq + Iii(x/xq), 



(20) 



where the new term hi(x/xo) is the dynamical counterpart 
of the information-cost of Ref. [4] . 
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3.3 Q-metric Brownian motion: the generalization to 
hyper-exponential growth 

We now relax the condition of proportional growth and 
appeal to the more general expression 



ry» I/* ryt^i 



(21) 



where q is a dimension-less parameter. It is easy to see that 
the two former examples are particular cases correspond- 
ing to q = (Brownian motion) and q — 1 (geometric 
Brownian motion). We call this new generalization of the 
dynamics the q-metric Brownian motion and proceed to a 
linearization of the dynamic equation by introduction of 
the variable 

u = log q (x/x ), (22) 

where \og q (z) is the q- logarithm of Tsallis' statistics [T3] . 
One finds 

u = k. (23) 

As before, this equation describes the Brownian motion in 
it, and thus a diffusion equation for a; of the form 



dt P (x,t) = Dd x {x q d x {x q p{x,t))), 



(24) 



whose solution for an initial Dirac-deltap(a;, 0) = 8 (x — x') 
is the q-log-normal distribution 



p{x,t) 



1 



(log^/rroJ-loggCx'/xo)) 2 



(25) 



y/AnDtxi 

We keep using Shannon's entropy in the MaxEnt approach: 



dup(u) logp(u), 



and the first constraint is expressed, as usual, via 



= 0, 



S — fi dup(u) 



(26) 



(27) 



which yields a constant density for u as p(u)du = Z~ 1 du, 
with Z = Q = \og q (xM/xo). Now, changing to the ob- 
servable x — Xq exp q (u) we obtain 



p{x)dx 



1 dx 
Zx?' 



(28) 



i.e., a power law. We express now the second constraint 
as (x) = xo(exp q (u)) — N/n c , writing 



S — /i / dup{u) — A / dup(u) exp (u) 



0, (29) 



which solution for its extremization is the density p(u)du — 
Z^ 1 exp(—Aexp q (u))du. Changing back to the observable 
x we now get 



p(x)dx = i eXp{ - Ax/Xo) dx. 



Z 



x.i 



(30) 





Fig. 3. Same as Fig. 1 for the q — 1.5 and q = 2 cases. 



The associated constants are then obtained from the con- 
straints and one sees that 



z = (A/ XQ )i- 1 r(i- q ,A) 
r(2~ qi A) = 

AT(1 - <j, A) n c x ' 



(31) 



We depict in Fig. 3 the distributions for the same param- 
eters as Figs. 1 and 2 for q = 1.5 and q = 2, obtaining 
A = 0.223742 and A = 0.103808, respectively. 
Assuming once more time that our constrain is a mean- 
energy-one, the effective Hamiltonian here reads [2] 



H = Ax/xq + q hi(x/xo). 



(32) 



Again, the new term resembles the information-cost of 
Ref. 4 . One can introduce once again an inverse "tem- 
perature" /? multiplying the Hamiltonian. The partition 
function Z = J dx cxp(~f3H) turns out to be expressed 
in terms of the incomplete Gamma function |12j 



Z=(J3A/x )to- 1 r(l-Pq,pA). 



(33) 



Comparing with Eq. ( |3l[ ) the partition function Z remains 
invariant save for a redefinition q — > f3q and A — > /3A. Note 
also that comparing with Eq. ( 19 ) we recover the propor- 



tional growth partition function at that special tempera- 
ture for which j3 = 1/q. Thus, by increasing T from zero to 
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Fig. 4. Diffusion of the initial delta distribution peaked at 
x = (x) = 2.5, at r =0.2 (blue), 1.5 (green) and 4 (red) MC 
steps, with the asymptotic distribution (black) compared with 
the MaxEnt prediction (black smooth line). Inset: convergence 
of the standard deviation for the q — case (see text). The 
shadowed area here reflects the numerical error 1 / ^/n^. 



q we can cancel out a dynamical behavior via "heating". 
Interestingly enough, in the limit f3 — > oo (T — > 0) the 
equilibrium density distribution is p(x)dx = S(x — x$)dx, 
i.e., all elements become placed at the same Xq. The ab- 
solute entropy vanishes, as it should (third law of ther- 
modynamics). Actual attainment of the T = 0— situation 
would entail a weird configuration indeed, since it seems 
impossible that xq could accommodate all elements simul- 
taneously. 



4 Numerical experiments and results 

4.1 Brownian motion 

We have proceeded to confirm our theoretical findings by 
means of numerical experiments, simulating the dynam- 
ics of random walkers following the dynamical equations 
proposed here. As a control case, we start with the lin- 
ear q — 0, largely used in physics in molecular dynamics, 
statistical mechanics and so on. Our algorithm works as 
follows: 

i) We firstly fix the minimum xq, the number of walkers 
n c and the mean value N/n c . We generate a vector x 
with n c elements as Xi = N/n c , Vi, which represents 
the walkers. 

iii) We randomly select the i-th walker and generate a new 
position by discretization of the dynamical equation as 
Xi = Xi + kdr, where k is a gaussian random number 
with variance K and zero mean, and dr is an arbitrary 
small 'time' interval. 

iv) We correct the mean value in a way compatible with 
the dynamical equation, i.e., linearly. A general ap- 
proach is via the change x' = x + A, where A — 
NJ n c — (x) . It is worth mentioning that the computa- 
tional time is reduced by randomly choosing a second 



1.0 




Fig. 5. Same as Fig. 5 for the q — 1 case, at r =0.2 (blue), 0.8 
(green) and 2.5 (red) MC steps. 



j-th walker and make it evolve with Xj — Xj — kdr 
using the same value of k as above. We finally accept 
the changes if, and only if, min(x) > x . 
v) We repeat iii) and iv) iteratively until achieving con- 
vergence in the distribution of x. 

We display in Fig. 5 the diffusion of n c — 100000 walkers 
initially peaked at x = 2.5 (N = 250000), for different 
simulation times, at 0.2, 1.5, and 4 Monte-Carlo steps, 
defining each MC step as n c iterations using K = 1 and 
dr = 10. We compare them with the asymptotic equilib- 
rium distribution (also shown in Fig. 1) and depict the 
convergence of the relative difference of the standard de- 
viation to what is predicted by our MaxEnt treatment, 
e(a) = |<7MaxEnt — fT ( T )l- As expected, after some steps 
we finally reproduce the theoretical MaxEnt distribution. 
We remark on the importance of correcting the positions 
of the walkers in linear fashion, respecting the dynamical 
equation and guaranteeing maintenance of the operative 
constraint at the main value (x). 

4.2 Geometric brownian motion 

We have proceeded for q — 1 in a similar way as in the 
precedent case, with the difference that now there are two 
equivalent descriptions for the dynamics involved. The al- 
gorithm for the first of them is as follows 
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i) We fix the values of xo, n c and N/n c . We again generate 
the walkers as a vector x with n c elements as Xj = 

N/n c , VI 

iii) We randomly select the i-th walker and generate a 
new position as Xj = x, + kxidr. 

iv) We now correct the mean value in a way compatible 
with the dynamical equation, i.e., proportionally. We 
change x' = x x A, where now A — (N/n c )/(x). We 
accept the changes if, and only if min(x) > Xo. 

v) We repeat iii) and iv) iteratively until encountering 
convergence in the distribution of x. 

This algorithm solves explicitly the equation of motion in 
x, which requires a specially small time interval dr in order 
to reduce the error in the discretization of the time deriva- 
tive. The convergence is achieved after a somewhat big 
computational effort since dr <C (^/Kxm) -1 ■ Wc highly 
recommend working with the variable u = log(x/xo) to 
linearize the equations, and apply afterwards the forth- 
coming algorithm: 

i) We fix the values of xo, n c , and N/n c to generate the 

walkers as a vector u with Ui = \og(N / (n c xo)) , Vz. 

ii) We randomly select the i-th walker and generate a new 
position as Ui = it* + kdr. 

iii) We now correct the mean value as u' = u + A, where 
now A = log [{N/n c )/ (xq exp(u))] — note that we use 
now the mean value of the exponential. We accept the 
changes if, and only if min(u) > 0. 

iv) We repeat ii) and iii) iteratively until reaching conver- 
gence in the distribution of u. 

It is easy to see that both algorithms are equivalents 
since Xj(l + kdr) ~ Xje fcdT = e Ul+kdT . We depict in Fig. 6 
the diffusion of the initial delta distribution at 0.2, 0.8 
and 2.5 MC steps using K = 1 and dr = 1 until getting 
convergence, with the same values for the parameters as in 
the previous instance. The final equilibrium distribution 
follows that predicted by MaxEnt, thus demonstrating the 
validity of our treatment. 

4.3 Q-metric brownian motion 

We finally describe the algorithm used for the general case. 
By recourse to the variable x: 

i) We fix the values of xq, n c , and N/n c to generate the 

vector x with Xi = N/n c , Vi. 

ii) We randomly select the i-th walker and generate a new 
position as x, = Xj + kx\dr. 

iii) We now correct the mean value as x' = x+x 9 Z\, where 
now A = (N/n c — (x))/(x q ), which guarantees obey- 
ing the dynamical equation — note that we explicitly 
recover the previous cases when q = and q = 1. We 
accept the changes if, and only if min(x) > xq. 

iv) We repeat ii) and iii) iteratively until convergence in 
the distribution of x. 

As in the q — 1 case, this algorithm demands, for a very 
small time interval dr, to reduce the error in the deriva- 
tive. We again recommend the use of the linearized vari- 
able u = log g (x/xo) via 
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Fig. 6. Top panels: same as Fig. 5 for the q = 1.5 case at r =0.1 
(blue), 0.5 (green) and 2 (red) MC steps. Bottom panels: q = 2 
case. 
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i) We fix the values of xo, n c , and N/n c to generate the 

vector u with = log g (iV/(n c a:o)), Vi. 

ii) We randomly select the i-th walker and generate a new 
position as it; = Ui + kdr. 

hi) We now correct the mean value as u' = u + Z\. Here A 
has no analytical expression, and is obtained from the 
equation (xoexp q (u + A)) — N/n c using an iterative 
Newton algorithm, reaching convergence in few steps. 
We accept the changes if, and only if min(u) > and 
max(u) < l/(q — 1). 

iv) We repeat ii) and iii) iteratively until obtaining con- 
vergence in the distribution of u. 

Using the same parameters as in the previous case, we de- 
pict if Fig. 7 the diffusion process at 0.1, 0.5 and 2 MC 
steps for the cases q = 1.5 and q — 2 respectively, also 
sowing the convergence of the standard deviation. System- 
atically, the equilibrium densities do follow the predicted 
distributions found via the MaxEnt. 
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5 Conclusions 

It is commonly believed that Jaynes' MaxEnt (JM), used 
in conjunction with Shannon's logarithmic information 
measure yields, after the concomitant variational process, 
only exponential probability distribution functions (PDF) . 
This fact was largely responsible for motivating statisti- 
cal mechanics' practitioners to look for other information 
measures [13]. We have shown here that great versatility 
is gained by MaxEnt if some further, appropriate a priori 
"dynamical" knowledge is added to the JM-technique, a 
way of proceeding that entirely agrees with Jaynes' phi- 
losophy |H2j . Indeed, we see that effective Hamiltonians 
for the process at hand are also a result of the MaxEnt 
technique. 

The JM-procedure can in this fashion still keep Shannon's 
measure while at the same yielding almost any functional 
form for the ensuing variational PDF, power laws in par- 
ticular. 
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